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Description 



Encoding apparatus, encoding method, decoding apparatus and 
decoding method 

Technical Field 

This invention relates to an encoding apparatus, encoding 
method, decoding apparatus and decoding apparatus, and for 
example, is suitably applied for transmitting image information 
(hereinafter, referred to as successive image information) 
composed of a plurality of unit image information being continued, 
via a network medium such as satellite broadcasting, cable TV or 
the Internet, or for processing successive image information on a 
storage medium such as an optical disc, a magnetic disk or a 
flash memory. 

Background Art 

Recently, for broadcasting stations and homes, such devices 
have been spread that adopt an encoding system such as MPEG 
(Moving Picture Experts Group) to encode (compress) successive 
images through orthogonal transformation such as discrete cosine 
transform, and motion compensation, utilizing the redundancy of 
the successive image information, for efficient information 
transmission or storage, by taking the successive images as 
digital information. 
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Especially, the MPEG2 (ISO/IEC 13818-2) encoding system is 
defined as a general image encoding system, and is widely used as 
an application for professionals and for consumers since it can 
treat interlaced images and progressively scanned images, and 
standard resolution images and high resolution images. By using 
this MPEG2 encoding system, high encoding efficiency (compression 
rate) and high quality of images can be provided, for example, by 
assigning interlaced images of standard resolution of 720 x 480 
pixels an amount of encoding (bit rate) of 4 to 8 [Mbps], or by 
assigning progressively scanned images of high resolution of 1920 
X 1088 pixels a bit rate of 18 to i22 [Mbps]. 

The MPEG2 encoding system is mainly used for encoding high 
quality of images for broadcasting and does not cope with an 
amount of encoding (bit rate) lower than that used by the MPEGl 
encoding system, that is, an encoding method with high encoding 
efficiency. It was expected that popularization of mobile 
terminals brings high needs of such an encoding system, and 
therefore the MPEG4 encoding system was standardized. The MPEG4 
encoding system for images was approved as international standard 
ISO/IEC 14496-2 in December 1998. 

In addition, recently, an encoding system (hereinafter 
referred to as JTV encoding system) called MPEG4AVC or H.264 was 
standardized by a joint video team composed of a VCEG group and 
an MPEG group. Compared with the MPEG2 and the MPEG4, this JVT 
encoding system can provide higher encoding efficiency although 
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it requires more operations for encoding and decoding. 

Now^ Fig. 8 shows a rough construction of an encoding 
apparatus which realizes an encoding process with any of the 
encoding systems referred above. As shown in Fig. 8^ the 
encoding apparatus 100 is composed of an image rearrangement 
buffer 102^ an adder 103, an orthogonal transformation unit 104, 
a quantization unit 105, a reverse encoding unit 106, a storage 
buffer 107, a dequantization unit 108, an inverse orthogonal 
transformation unit 109, a frame memory 110, a motion 
prediction/compensation unit 111 and a rate control unit 112. 

In this case, the encoding apparatus 100 stores successive 
image information in the image rearrangement buffer 102 to 
rearrange the successive image information according to GOP 
(Group of Pictures) structure on a unit-image-information basis 
(frame by frame or field by field). 

The image rearrangement buffer 102 gives the orthogonal 
transformation unit 104 unit image information out of the 
successive image information, which should be intra-prediction- 
encoded. The orthogonal transformation unit 104 applies 
orthogonal transformation such as the discrete cosine transform 
or the Karhunen Loeve transform, to the unit image information 
and gives an obtained orthogonal transformation coefficient to 
the quantization unit 105. 

The quantization unit 105 performs a quantization process 
on the orthogonal transformation coefficient given from the 
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orthogonal transformation unit 104, under the control of the rate 
control unit 112, and supplies obtained quantized information (a 
quantized orthogonal transformation coefficient) to the reverse 
encoding unit 106 and the dequantization unit 108. The reverse 
encoding unit 106 applies variable-length coding or reverse 
encoding such as arithmetic coding to the quantized information, 
and stores obtained encoded information (encoded quantized- 
information) in the storage buffer 107. 

The dequantization unit 108 applies a dequantization 
process to the quantized information and supplies obtained 
orthogonal transformation coefficient to the inverse orthogonal 
transformation unit 109. The inverse orthogonal transformation 
unit 109 applies the inverse orthogonal transformation to the 
orthogonal transformation coefficient and stores, if necessary^ 
obtained unit image information in the frame memory 110 as 
reference image information. 

On the other hand, the image rearrangement buffer 102 
supplies unit image information which should be inter-prediction- 
encoded, out of the successive image information to the motion 
prediction/compensation unit 111. The motion 
prediction/compensation unit 111 performs a motion 
prediction/compensation process by using the unit image 
information and reference image information read from the frame 
memory 10, and supplies obtained predicted image information to 
the adder 103. The adder 103 supplies to the orthogonal 
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transformation unit 104 difference between the predicted image 
information and corresponding unit image information as 
differential information • 

This differential information is subjected to various 
processes, as in the case of the intra-encoding, and the 
resultant is stored in the storage buffer 107 as encoded 
information and is stored, if necessary, in the frame memory 110 
as reference image information. 

In addition, the motion compensation/prediction unit 111 
gives the reverse encoding unit 106 motion vector information 
which is obtained together with the predicted image information 
as a result of the motion prediction/compensation process. The 
reverse encoding unit 106 performs the reverse encoding process 
on the motion vector information to thereby obtain encoded motion 
vector information for the header part of the corresponding 
encoded information . 

In such a manner, the encoding apparatus 100 successively 
creates encoded information on a unit-image-information basis by 
performing the encoding process on the successive image 
information, and successively outputs the encoded information via 
the storage buffer 107. 

Next, Fig. 9 shows a rough construction of a decoding 
apparatus which performs a decoding process corresponding to the 
encoding system of the encoding apparatus 100. As shown in Fig. 
9, the decoding apparatus 120 is composed of a storage buffer 121, 
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a reverse decoding unit 122, a dequantization unit 123, an 
inverse orthogonal transformation unit 124, an adder 125, an 
image rearragement buffer 126, a motion prediction/compensation 
unit 127, and a frame memory 128. 

In this case, the decoding apparatus 120 temporarily stores 
encoded information which is successively inputted, in the 
storage buffer 121 and then supplies it to the reverse decoding 
unit 122. In a case where the encoded information have been 
subjected to the intra-prediction encoding, the reverse decoding 
unit 122 applies a decoding process, variable-length decoding or 
arithmetic decoding, to the encoded information, and supplies 
obtained quantized information to the dequantization unit 123. 

The dequantization unit 123 applies a dequantization 
process to the quantized information given from the reverse 
decoding unit 122 and supplies obtained orthogonal transformation 
coefficient to the inverse orthogonal transformation unit 124. 
The inverse orthogonal transformation unit 124 applies an inverse 
orthogonal transformation process to the orthogonal 
transformation coefficient to thereby create the original image 
information before the encoding process (hereinafter, referred to 
as restored image information), and stores this in the image 
rearrangement buffer 126. 

On the other hand, in a case where the encoded information 
have been subjected to the inter prediction encoding, the reverse 
decoding unit 122 performs a decoding process on both of this 
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encoded information and the encoded motion vector information 
which has been inserted in the header part of the encoded 
information r and supplies obtained quantized information to the 
dequantization unit 123 and supplies the motion vector 
information to the motion prediction/compensation unit 127. The 
quantized information is subjected to various processes, as in 
the case of decoding encoded information intra-encoded, and is 
then supplied to the adder 125 as dif f erentiial information. 

In addition, the motion prediction/compensation unit 127 
creates predicted image information based on the motion vector 
information and reference image information stored in the frame 
memory 128, and supplies this to the adder 125. The adder 125 
synthesizes the reference image information and the differential 
information, and stores obtained restored image information to 
the image rearrangement buffer 126. 

In the aforementioned manner, the decoding apparatus 120 
successively creates restored image information by performing the 
decoding process on the encoded information successively inputted, 
and successively outputs the restored image information via the 
image rearrangement buffer 126 to, for excunple, a display unit 
(not shows) for successive reproduction. 

By the way, since the MPEG2 encoding system prescribes that 
only I (Intra) -pictures and P (Predictive) -pictures are used as 
pictures to be inter-prediction-encoded, a decoding order for the 
decoding process is naturally determined. 
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Therefore, in a case where the decoding apparatus 12 0 
successively reproduces restored image information which is 
created by performing the decoding process on encoded information 
with the MPEG 2 encoding system, it can appropriately display 
images based on the restored image information on the display 
unit without adjusting the output timing of the restored image 
information at the image rearrangement buffer 126. 

On the other hand, as compared with the MPEG2 encoding 
system, the JVT encoding system has a larger degree of freedom 
for selection of pictures to be prediction-encoded, for example, 
it can treat not only I- and P-pictures but also B 
(Bidirectional) -pictures as pictures to be inter-prediction- 
encoded. 

However, the JVT encoding system does not prescribe a 
decoding order in a decoding process and further does not specify 
output timing of restored image information. 

Therefore, if the decoding apparatus 120 successively 
reproduces restored image information which is created by 
performing the decoding process on encoded information with the 
JVT encoding system, such happening occurs that encoded 
information is still being decoded at the output timing for 
restored image information corresponding to the encoded 
information due to a limited resource of the image rearrangement 
buffer 126, and as a result, the continuousness is broken. 
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Disclosure of the Invention 

This invention was made in view of the above points and 
intends to an encoding apparatus and encoding method for making a 
decoding apparatus perform successive reproduction^ and to a 
decoding apparatus capable of performing successive reproduction 
and a decoding method. 

To solve such problems, in this invention, an encoding 
apparatus which carries out an encoding process with an encoding 
system capable of treating at least B-pictures as pictures to be 
prediction-encoded comprises a timing calculation means for 
calculating output timing of results of decoding a plurality of 
encoded information, anticipating that the encoded information 
created through the encoding process will be successively decoded 
on a decoding side, and a timing notification means for notifies 
the decoding side of each of the output timing calculated by the 
timing calculation means before a result of decoding 
corresponding encoded information is obtained. 

Therefore, the encoding apparatus of this invention can 
make the decoding side recognize output timing calculated 
assuming that encoded information will be decoded on the decoding 
side, so as to keep the output continuousness of restored image 
information even the encoded information have been subjected to 
encoding with an encoding system which does not allow a decoding 
order to be naturally determined, thus the decoding side can 
perform continuous reproduction. 
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Further, in an encoding method for performing an encoding 
process with an encoding system capable of treating at least B- 
pictures as pictures to be prediction-encoded , it is anticipated 
that a plurality of encoded information created by performing the 
encoding process will be successively decoded on a decoding side, 
and output timing of results of decoding the encoded information 
are calculated and the decoding side is notified of the 
calculated output timing before a result of decoding 
corresponding encoded information is obtained. 

Therefore, in the encoding method according to this 
invention, by making the decoding side recognize output timing 
calculated anticipating that encoded information will be decoded 
on the decoding side, the output continuousness of restored image 
information can be kept even the encoded information have been 
subjected to encoding with an encoding system which does not 
allow a decoding order to be naturally determined, thus the 
decoding side can perform continuous reproduction. 

Still further, a decoding apparatus for performing a 
decoding process on a plurality of encoded information which have 
been subjected to encoding with an encoding system capable of 
treating at least B-pictures as pictures to be prediction-encoded 
comprises a storage means for temporarily storing restored image 
information successively created as a result of the decoding 
process, and an output control means for controlling output of 
the restored image information to be stored in the storage means. 
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If restored image information to be stored in the storage means 
is failed, the output control means re-outputs restored image 
information outputted just before the failure. 

Therefore, the decoding apparatus of this invention can 
keep the output continuousness of restored image information even 
encoded information have been subjected to encoding with an 
encoding system which does not allow a decoding order to be 
naturally determined, and thus can perform continuous 
reproduction . 

Still further, in this invention, a decoding method for 
performing a decoding process on a plurality of encoded 
information which have been subjected to encoding with an 
encoding system capable of at least B-pictures as pictures to be 
prediction-encoded comprises a first step of temporarily storing 
restored image information sequentially created as a result of 
the decoding process, a second step of outputting the restored 
image information to be stored, and a third step of, in a case 
where restored image information to be stored has been failed, 
re-outputting restored image information outputted just before 
the failure. 

Therefore, in the decoding method of this invention, even 
encoded information have been subjected to encoding with an 
encoding system which does not allow a decoding order to be 
naturally determined, the output continuousness of restored image 
information can be kept, thus continuous reproduction can be 
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performed. 



Detailed Description of the Drawings 

Fig._LJ-S a block diagram showing an image reproduction 
system according to the first embodiment. 

Fig . 2 i s a functional block diagram showing the processing 
contents of an encoding control unit. 

Fig. 3 is a table used for explaining calculation of output 
timing. 

Fig. 4 is a flowchart showing a procedure for an output 
timing notification process. 

Fig . ^5 i s a block diagram showing an image reproduction 
system according to the second embodiment. 

Fig. 6 is a table used for explaining control of re-output. 

Fig. 7 is a flowchart showing a procedure for a re-output 
control process. 

Fig. 8 is a block diagram showing the construction of an 
encoding apparatus. 

Fig. 9 is a block diagram showing the construction of a 
decoding apparatus. 

Best Mode for Carrying Out the Invention 
Embodiment 1 

( 1 ) Construction of Image Reproduction System 1 

Referring to Fig. 1, reference numeral 1 shows an image 
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reproduction system according to the first embodiment as a whole, 
which is constructed by connecting an encoding apparatus 2 with 
the JVT encoding system and a decoding apparatus 3 to each other 
with a prescribed transmission line. 

The encoding apparatus 2 includes an encoding unit 10 
having the same construction as the encoding apparatus 100 
described with reference to Fig. 8 and an encoding control unit 
11 for controlling the encoding unit 10, and similarly to the 
case described with reference to Fig. 8, performs an encoding 
process on successive image information Dl through the encoding 
unit 10 controlled by the encoding control unit 11 to 
successively create encoded information D2 (D2a, D2b, ... or D2n) 
from unit image information (frame data or field data) Dla to Din, 
and then outputs the encoded information D2, the successive image 
information Dl being supplied from the outside or being read from 
an internal recording medium (not shown) such as an HDD (Hard 
Disk Drive) . 

The decoding apparatus 3, on the other hand, includes a 
decoding unit 20 having the same construction as the decoding 
apparatus 120 described with reference to Fig. 9 and a decoding 
control unit 21 for controlling the decoding unit 20, and 
similarly to the case described with reference to Fig. 9, 
performs a decoding process on the encoded information D2 
successively inputted through the transmission line, through the 
decoding unit 20 controlled by the decoding control unit 21 to 



13 



successively create restored image information D3 (D3a, D3b, ... 
or D3n), and then successively outputs the restored image 
information D3 to a display unit (not shown), which results in 
continuous reproduction. 

(2) Construction of Encoding Control Unit 11 

This encoding control unit 11 of the encoding apparatus 2 
detects the GOP structure of the successive image information Dl 
inputted to the encoding unit 10 and conditions on an encoding 
process, such as an encoding order, (hereinafter, referred to as 
encoding conditions), based on previously stored programs, table 
information and so on with the JVT encoding system, to control 
the encoding unit 10 on the encoding conditions. 

In addition to the above structure, the encoding control 
unit 11 calculates output timing of the restored image 
information D3 which is a result of decoding the encoded 
information D2, anticipating that the encoded information D2 will 
be successively decoded on a decoding side (decoding apparatus 3), 
and performs an output timing notification process to notify the 
decoding apparatus 3 of this output timing before the restored 
image information D3 is created. 

Now, if the processing contents of the output timing 
notification processing by the encoding control unit 11 is 
divided functionally, they can be divided into a delay 
calculation unit 11a for calculating a period of time after a 
decoding process of the encoded information D2 is started until 
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the decoded image information D3 created by the process is 
outputted (hereinafter, the period of time is referred to as 
decode delay), and a header addition unit lib for adding a decode 
delay as the header of corresponding encoded information D2, as 
shown in Fig. 2. Processes by the delay calculation unit 11a and 
the header addition unit will be described by using an example 
shown in Fig. 3. 

Referring to Fig. 3, a column ''^EI (Encoder Input)" shows 
picture types which are assigned to unit image information Dla to 
Din to be inputted to the encoding unit 10, according to the GOP 
structure, that is, a picture-type order before the encoding 
process. A column ^'EO (Encoder Output)" shows the picture types 
of the encoded information D2a to D2n created by performing the 
encoding process on the unit image information Dla to Din 
inputted to the encoding unit 10 in a prescribed encoding order, 
that is, a picture-type order after the encoding process. 

A column ^^Ed (Encoder Delay)" shows a period of time after 
the encoding process of unit image information Dla, Dlb, ... or 
Din is started until the encoded information D2 created by the 
process is outputted (hereinafter, this period of time is 
referred to as an encode delay) and the period of time is 
calculated based on the encoding conditions. Based on encode 
delays (in a column ^^Ed"), output timing of the encoded 
information D2a to D2n (in a column ^^EO") are adjusted if 
necessary. Specifically, as to the first unit image information 
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Dla C'lOO" in the column "'EI") which is of an I-picture type, for 
example, if a period of time after it is inputted (t=0) until it 
is converted into the encoded information D2a (^^100'' in the 
column ^'EO") is shorter than a corresponding encode delay (six 
seconds in the column ^'Ed"), its output timing is adjusted by 
storage in the storage buffer 107 (Fig. 8) of the encoding unit 
10 and the information is outputted when the encode delay is 
passed ( t=6 ) . 

The delay calculation unit 11a calculates decode delays (in 
a column '^Dd (Decoder Delay)") for the encoded information 02 
based on the encode delays, and calculates the decode delays of 
the encoded information D2a to D2n so that a decode delay of 
encoded information D2 (''B13") having the longest encode delay 
(10 seconds) is the shortest (0 second). 

Specifically, since the longest encode delay is 10 
[seconds], the delay calculation unit 11a subtracts the value of 
the. encode delay corresponding to each piece of encoded 
information D2a to D2n from 10 [seconds] to thereby calculate the 
values of decode delays (values in the column ^'Dd") for the 
encoded information D2a to D2n. 

Then, the delay calculation unit 11a creates decode delay 
information DIO from thus calculated decode delays and sends them 
to the header addition unit lib. 

Every time when the encoding unit 10 creates encoded 
information D2, the header addition unit lib adds, based on the 
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decode delay information DIO supplied from the delay calculation 
unit 11a, a corresponding decode delay to the encoded information 
D2 by placing the delay in the header. 

In this case, for example, the encoded information D2a 
C^IOO'' in the column '"EO") having such a decode delay added as 
the header is outputted when the encode delay is passed and is 
inputted into the decoding apparatus 3 through the transmission 
line. Note that, in Fig. 3, the transmission time through the 
transmission line between the encoding apparatus 2 and the 
decoding apparatus 3 is not considered, and therefore the input 
timing of encoded information D2 to the decoding apparatus 3 is 
taken as the same as the output timing (in the column '"EO") of 
the encoded information D2 from the encoding apparatus 10. 

The decoding apparatus 3 adjusts the output timing of 
restored image information D3 which is a result of decoding 
encoded information D2, based on the decode delays of the encoded 
information D2. That is, the decode delay (4 [seconds] in the 
column ^^Dd'') of the first encoded information D2a, for example, 
is recognized by the decoding control unit 21 of the decoding 
apparatus 3 based, on the header added to the encoded information 
D2a before the encoded information D2a is decoded. Then, the 
encoded information D2a is converted into restored image 
information D3a by the decoding unit 20, and if a passage time 
from the input (t=6) is shorter than the corresponding encode 
delay (4 [seconds]), its output timing is adjusted by storage in 
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the image rearrangement buffer 126 (Fig. 9) according to 
necessity, and the restored image information D3a is outputted 
when the decode delay (4 [seconds] (t=10)) is passed, to be 
displayed on the display unit. 

In addition, as to the other encoded information D2b to D2n, 
similar to the encoded information D2a, they are converted into 
the restored image information D3b to D3n by the decoding unit 20, 
and these restored image information D3b to D3n are rearranged, 
if necessary, by the image rearrangement buffer 126 (Fig. 9) to 
have the same picture-type order before the encoding process (in 
a column ^^DO (Decoder Output)"), and are outputted after the 
output timing are adjusted based on corresponding encode delays, 
which results in continuous reproduction. 

The encode control unit 11 performs the output timing 
notification process as described above, to thereby make the 
decoding apparatus 3 recognize decode delays via headers, the 
decode delays calculated assuming that encoded information D2 
will be decoded by the decoding apparatus 3. 

Now, the output timing notification process as described is 
carried out following a procedure RTl for the output timing 
notification process shown in Fig. 4. 

Specifically, the encoding control unit 11 starts the 
procedure RTl for the output timing notification process from 
step SPO when predetermined operations to execute the encoding 
process are performed with an input unit (not shown), and finds 
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out the longest encode delay (which corresponds to 10 [seconds] 
in the column ^^Ed" in Fig. 3) out of encode delays calculated on 
the encoding conditions at following step SPl. 

Then the encoding control unit 11 stores, for example, the 
encoded information D2a existing in the reverse encoding unit 106 
(Fig, 8) of the encoding unit 10, in the storage buffer 107 (Fig. 
8) at step SP2, and judges at step SP3 whether the storage has 
been done successfully, and if a negative result is obtained, the 
process returns back to step SP2 to re-store the encoded 
information D2a. 

If an affirmative result is obtained at step SP2, on the 
contrary, the encoding control unit 11 subtracts the encode delay 
(which corresponds to 6 [seconds] in the column ^^Ed'' in Fig. 3) 
for the encoded information D2a stored at step SP2 from the 
encode delay recognized at step SPl, to thereby calculate a 
decode delay (which corresponds to 4 [seconds] in the column 
^'Dd") for the encoded information D2a, and adds the calculated 
decode delay to the encoded information D2a as a header at step 
SP4. 

Then, the encoding control unit 11 judges at step SP5 
whether all of the encoded information D2 have been talcen in the 
storage buffer 107 (Fig. 8), and if a negative result is obtained, 
it returns baclc to step SP2 and repeats the above processes, and 
on the contrary, if an affirmative result is obtained, it moves 
on to step SP6 where the procedure RTl for the output timing 
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notification process is terminated. 

As described above, the encoding control unit 11 can 
execute the output timing notification process following the 
procedure RTl for the output timing notification process. 

In the aforementioned construction, the encoding apparatus 
2 calculates output timing (decode delay) for a result (restored 
image information D3) of decoding the encoded information D2, 
anticipating that the encoded information D2 obtained as a result 
of encoding with the JVT encoding system will be decoded on the 
decoding apparatus 3 side, and adds the calculated output timing 
as a header. 

Therefore, this encoding apparatus 2 can make the decoding 
apparatus 3 recognize via headers decode delays which are 
obtained assuming that the encoded information D2 will be decoded 
by the decoding apparatus 3, before the decoding process, so that 
even the encoded information D2 have been encoded with the JVT 
encoding system which does not allow the decoding order to be 
naturally determined, the output continuousness of restored image 
information D3 can be kept. 

In this case, the encoding apparatus 2 calculates output 
timing (decode delays) for restored image information D3 so as to 
immediately output a result of decoding encoded information D2 
('^B13" in Fig. 3) having the longest periods of time out of 
periods of time after the encoding process is started till 
encoded information D2 is outputted. 
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Therefore, the encoding apparatus 2 can calculate output 
timing (decode delays) for results (restored image information 
D3 ) of decoding encoded information D2 based on output timing 
(encode delays) from the decoding side for encoded information D2 
which needs the longest time to be decoded on the decoding side, 
and as a result, the output timing from the decoding apparatus 3 
can be adjusted (by offset) so as not to occur underflow. 

According to the above construction, it is assumed that 
encoded information D2 obtained through encoding with the JVT 
encoding system will be decoded on the decoding apparatus 3 side, 
and output timing (decode delays) for restored image information 
D3 obtained by decoding the encoded information D2 are calculated 
and the calculated output timing are added as headers, and 
thereby even the encoded information D2 have been encoded with 
the JVT encoding system which does not allow the decoding order 
to be naturally determined, the output continuousness of the 
restored image information D3 can be kept, thus the decoding 
apparatus 3 can perform continuous reproduction. 
(3) other Embodiments 

Note that, the aforementioned first embodiment has 
described the case where the JVT encoding system is applied. 
This invention, however, is not limited to this and another kind 
of encoding system which can treat at least B-pictures as 
pictures to be prediction-encoded can be applied. 

Further, the aforementioned first embodiment has described 
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the case where a timing calculation means for, assuming that a 
plurality of encoded information D2 created by performing the 
encoding process are sequentially decoded on a decoding side, 
calculating output timing for results of decoding the encoded 
information calculates decode delays for results of decoding the 
encoded information so that a result of decoding encoded 
information D2 having the longest encode delay out of the encode 
delays is immediately butputted. This invention, however, is not 
limited to this and a period of time after a result of decoding 
encoded information is obtained until its output timing may be 
calculated instead of the decode delays, or a decode delay for a 
result of decoding encoded information may be calculated based on 
a calculation result on an assumed occupation rate of information 
stored in a buffer of the decoding side. If output timing can be 
calculated based on such a calculation result, such output timing 
can be calculated as to make the decoding side perform stable 
continuous reproduction. 

Further, the aforementioned first embodiment has described 
the case where a timing notification means for notifying a 
decoding side of output timing before a result of decoding 
corresponding encoded information is obtained adds output timing 
(decode delay) to corresponding encoded information D2 as its 
header based on decode delay information DIO supplied from the 
delay calculation unit 11a. This invention, however, is not 
limited to this and the decode delay information DIO can be 
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directly outputted to the decoding side before an encoding 
process, without being added as a header. 

Embodiment 2 

(1) Construction of Image Reproduction System 51 

In Fig. 5 in which the same reference numerals are applied 
to those of corresponding parts in Fig. 1, reference numeral 51 
shows an image reproduction system according to the second 
embodiment as a whole and the system is constructed by connecting 
an encoding apparatus 52 with the JVT encoding system and a 
decoding apparatus 53 to each other with a prescribed 
transmission line • 

The encoding apparatus 52 has an encoding unit 10 and an 
encoding control unit 61 for controlling the encoding unit 10, 
and the encoding control unit 61 does not perform the 
aforementioned output timing notification process but carries out 
the other processes performed in the aforementioned first 
embodiment. In this case, the encoding apparatus 52 performs an 
encoding process on successive image information Dl with the 
encoding unit 10 controlled by the encoding control unit 61, to 
thereby successively create encoded information D2 without a 
decode delay added as a header, and then successively outputs the 
encoded information D2 . 

The decoding apparatus 53, on the other hand, has a 
decoding unit 20 and a decoding control unit 71 for controlling 
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the decoding unit 20, and performs a decoding process on the 
encoded information D2 successively inputted through the 
transmission line, through the decoding unit 20 controlled by the 
decoding control unit 71, to thereby create restored image 
information D3, and then successively outputs the restored image 
information D3 to a display unit (not shown) for successive 
reproduction. 

(2) Construction of Decoding Control Unit 71 

Actually, the decoding control unit 71 of this decoding 
unit 53 temporarily stores successively inputted encoded 
information D2 in a storage buffer 121 (Fig. 9) of the decoding 
unit 20, detects conditions on decoding (hereinafter referred to 
as decoding conditions ) , such as a decoding order of the encoded 
information D2 and a start time of a decoding process, based on 
the headers of the encoded information D2 , and thus can control 
the decoding unit 20 on the decoding conditions. 

In addition to the above structure, the decoding control 
unit 71 watches a storage state of the restored image information 
D3 stored in an image rearrangement buffer 126 (Fig. 9) after the 
decoding process, and when detecting underflow as the storage 
state, carries out a re-output control process for re-out putting 
the restored image information D3 outputted just before the 
detection (underflow). This re-output control process will be 
described by using an example shown in Fig. 6. 

In this Fig. 6, as in the case of Fig. 3, a column ^^EI" 



24 



shows a picture-type order before an encoding process, a column 
^'EO" shows a picture-type order after the encoding process, a 
column '^DC shows a picture-type order after decoding, a column 
'"Ed" shows encode delays, and a column ^'Dd" shows decode delays. 
In addition, as in the case of Fig. 3, in this Fig. 6, a 
transmission time through the transmission line between the 
encoding apparatus 52 and the decoding apparatus 53 is not 
considered and an input time of encoded information D2 to the 
decoding apparatus 53 is taken to the same as an output time (in 
the column ^^EO") of the encoded information D2 from the encoding 
unit 10. 

In addition, a column ^^Sud (Start-up delay)'' shows a period 
of time after each piece of encoded information D2 is inputted 
until a decoding process is started (hereinafter, this period of 
time is referred to as a start-up delay), and this start-up delay 
is calculated based on the encoding conditions by the encoding 
control unit 61 of the encoding apparatus 52 and is added as a 
header. 

When the decoding control unit 71 receives the first 
encoded information D2a (^^100" in the column ^'EC), it ignores 
the start-up delay for the encoded information D2a (^^6" [seconds] 
in the column ^^Sud") and performs control to immediately send the 
encoded information D2a to the reverse encoding unit 122 and 
start a decoding process (^^0" [second] in the column ^^Dd"). As a 
result, the decoding control unit 71 can shorten a preparation 
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time (driving time) for continuous reproduction. 

In this case, the decoding control unit 71 has a lag 
between the actual start time of the decoding process and the 
original start time, due to the control (this lag has to be 
adjusted and specifically, is 6 [seconds] in Fig, 6 and is 
hereinafter referred to as a delay amount to be adjusted), so 
that restored image information D3a which is a result of decoding 
the encoded information D2a is outputted via the image 
rearrangement buffer 126 and then re-outputted (at an item ^^t=8" 
in the column ^^DO") so as to offset a part of the delay eonount to 
be adjusted (which corresponds to ^^2'' [seconds] of ^^t=8" in the 
column ^'Dd'') . 

Then, the decoding control unit 71 have encoded information 
D2b, D2c, successively inputted following the encoded 

information D2a ('"POl", ''P02", in the column "'EC'), decoded 

according to the start-up delays, and manages the storage state 

of the restored image information D3b, D3c, (""POl", ''P02", 

in the column ^'DO") being stored in the image rearrangement 
buffer 126 (Fig. 9) after the decoding process. 

Under this state, every time when restored the image 
information D3 to be stored in the image rearrangement buffer 126 
is failed (underflow), the decoding control unit 71 re-outputs 
the restored image information D3 outputted just before the 
failure (''P05" and "^PIO" in the column ""DO"), to thereby 
periodically offset the other delay amounts to be adjusted. 
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In such a manner, the decoding control unit 71 executes the 
re-output control process, so as to keep the output 
continuousness of the restored image information D3. 

Now, the output timing notification process described above 
is sequentially executed following a procedure RT2 for the re- 
output control process shown in Fig. 7. 

That is, when predetermined operations to execute a 
decoding process are performed with an input unit (not shown) for 
example, the decoding control unit 71 starts the procedure RT2 
for the re-output control process from step SPIO, and waits for 
the first encoded information D2a to be inputted at next step 
SPll, and when receiving the encoded information D2a, moves on to 
step SP12. 

Then, the decoding control unit 71 ignores a start-up delay 
for the encoded information D2a and immediately starts the 
decoding process at step SP12, and after outputting restored 
image information D3a obtained as a result of the decoding 
process at next step SP13, re-outputs the restored image 
information D3a at step SP14 to offset a part of a delay amount 
to be adjusted. 

Next, the decoding control unit 71 starts decoding of 
following restored image information D2 (D2b, D2c, ... D2n) and 
outputs restored image information D3 (D3b, D3c, ... D3n) 
obtained by the process at step SP15, and at next step SP16 
judges whether restored image information D3 to be stored in the 
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image rearrangement buffer 126 (Fig. 9) has been failed, and then 
if a negative result is obtained, returns back to step SP15 and 
repeats the above processes. 

If an affirmative result is obtained, on the contrary, the 
decoding control unit 71 re-outputs the restored image 
information D3 (^^P05" and "^PlO" in the column '^DO'') outputted at 
step SP15 to offset a part (or all) of the remaining delay amount 
to be adjusted and moves on to next step SP18. 

Then, the decoding control unit 71 judges at step SP18 
whether all of the delay amount to be adjusted is offset, and if 
a negative result is obtained, returns back to step SP15 and 
repeats the above processes. And if an affirmative result is 
obtained, on the contrary, the decoding control unit 71 moves on 
to step SP19 where this procedure RT2 for the re-output control 
process is terminated. 

In such a manner, the encoding control unit 11 can carries 
out the re-output control process following the procedure RT2 for 
the re-output control process. 

According to the above construction, this decoding 
apparatus 53 temporarily stores the restored image information D3 
successively created by performing the decoding process on the 
encoded information D2 which have been encoded with the JVT 
encoding system, in the image rearrangement buffer 126 (Fig. 9), 
and if restored image information D3 to be stored in the image 
rearrangement buffer 126 (Fig. 9) is failed, re-outputs the 
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restored image information D3 outputted just before the failure. 

Therefore, the decoding apparatus 53 can keep the output 
continuousness of the restored image information D3 even the 
encoded information D2 have been subjected to encoding with the 
JVT encoding system which does not allow a decoding order to be 
naturally determined. 

In this case, the decoding apparatus 53 ignores the 
decoding start time (start-up delay) set for the first encoded 
information D2 stored in the storage buffer 121 (Fig. 9) and 
immediately starts decoding of the encoded information D2a, and 
when restored image information D3 to be stored in the image 
rearrangement buffer 126 (Fig. 9) is failed, offsets a lag (delay 
amount to be adjusted) from the set decoding start time occurred 
due to the ignorance, by re-outputting the restored image 
information outputted just before the failure. 

As a result, the decoding apparatus 53 can shorten a 
preparation time (driving time) for continuous reproduction and 
can keep the output continuousness of restored image information 
while periodically dispersing a lag occurred due to the 
shortening (delay amount to be adjusted). 

According to the aforementioned construction, the restored 
image information D3 sequentially created by performing the 
decoding process on the encoded information D2 which have been 
subjected to encoding with the JVT encoding system are 
temporarily stored in the image rearrangement buffer 126 (Fig. 9), 
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and when restored image information D3 to be stored in the image 
rearrangement buffer 126 (Fig. 9) is failed, the restored image 
information D3 outputted just before the failure is re-outputted, 
so that even the encoded information D2 have been subjected to 
encoding with the JVT encoding system which does not allow a 
decoding order to be naturally determined, the output 
continuousness of the restored image information D3 can be kept, 
which results in the continuous reproduction. 
(3) Other Embodiment 

Note that, the aforementioned second embodiment has 
described the case where the JVT encoding system is applied. 
This invention, however, is not limited to this and another kind 
of encoding system which can treat at least B-pictures as 
pictures to be prediction-encoded can be applied. 

Further, the aforementioned second embodiment independently 
uses the storage buffer 121 and the rearrangement buffer 126 as 
storage means for temporarily storing encoded information and 
restored image information sequentially created by performing a 
decoding process on the encoded information. This invention, 
however, is not limited to this and the storage buffer 121 and 
the rearrangement buffer 126 are used in common. In this case, 
the number of buffers and manner of storage can be changed 
according to necessity. 

Still further, the aforementioned second embodiment has 
described the case where the decoding control unit 71 serving as 
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an output control means ignores a decoding start time (start-up 
delay) set for the first encoded information D2a being stored in 
the storage buffer 121 (Fig. 9) and immediately starts decoding 
of the encoded information D2a, and when restored image 
information D3 to be stored in the image rearrangement buffer 126 
(Fig. 9) is failed, offsets a lag (a delay amount to be adjusted) 
from the set decoding start time occurred due to the ignorance, 
by re-outputting restored image information outputted just before 
the failure. In addition to this, if a storing order (in the 
column ^^EO") of any of the encoded information D2a to D2b stored 
in the storage buffer 121 is different from an order before 
encoding (in the column ^'EI'') (for example, in the item ^^t=32" in 
Fig. 6), the restored image information D3 corresponding to the 
encoded information D2 having the different order may be re- 
outputted. By doing so, a decode delay which becomes longer when 
a different order is generated can be filled by re-output, which 
results in more assured output continuousness of the restored 
image information D3. 

Industrial Applicability 

This invention can be used for a case of transmitting 
successive image information via a network medium such as 
satellite broadcasting, cable TV or the Internet, or a case of 
processing the successive image information on a storage medium 
such as an optical disc, magnetic disk or flash memory. 
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